6.4. Platform Tools

Which component serves the tools?

On your laptop the agent started the tool server itself, as a child process. In the cluster the tool server is its own pod, and the agent only knows its URL.

The code is unchanged. infra/k8s/base/mcp.yaml runs the same agentops-agent:dev image with a different entrypoint, so the tool code and the agent code ship as one artifact but run as two workloads:

command: ["python", "-m", "agent.mcp_server"]
env:
  - name: MCP_HOST
    value: 0.0.0.0
  - name: MCP_PORT
    value: "8000"
  - name: MCP_TRANSPORT
    value: streamable-http

This is the in-cluster form of the streamable-HTTP transport 3.3. MCP already built. One line each:

1. MCP_HOST=0.0.0.0 binds all interfaces.
2. MCP_PORT=8000 matches the service.
3. streamable-http is the stateless (stateless_http=True) transport, so any replica could answer any request without session affinity.

The deployment also sets AGENT_STATE_DIR=/app/state and AGENT_DATA_DIR=/app/data — the runtime database mount and the immutable image seed — which the coherence question below depends on.

A single ClusterIP service, Service/agentops-mcp, exposes port 8000 inside the namespace. No LoadBalancer, NodePort, or Ingress is created.

Why run the tools as a separate in-cluster deployment?

Promoting the tools to their own Deployment buys four things the in-process form cannot.

The host form is zero-friction for a first run and nothing more. There, MCP runs as a stdio child the agent spawns, which couples tool availability to launching a subprocess on every agent host and leaves nowhere to enforce policy.

Each of the four gains maps to a concrete property of mcp.yaml:

1. Independent lifecycle and scaling. The tool pod restarts, drains (a 15 s terminationGracePeriodSeconds that exceeds the Uvicorn shutdown window), and rolls without touching the agent, and the reverse holds too. A wedged tool process is one pod, not the whole agent.
2. A read-only state mount. The tools mount the state claim readOnly: true, so the read service physically cannot write the database the agent owns.
3. Network isolation. A distinct pod is a distinct NetworkPolicy subject, so only agentgateway may dial port 8000.
4. One shared MCP surface. The same MCPServer backs the ADK agent and any other MCP client. 3.3. MCP is the general reuse argument; in the cluster the second client is kagent, below.

Which six tools does the server expose?

Exactly six read capabilities cross the MCP boundary, and naming them is what turns "read-only" into a checkable claim rather than a slogan.

mcp_server.py re-exposes four operational tools from tools.py:

1. list_incidents — incidents on the platform, most recent first.
2. get_incident — the full details of one incident by its id.
3. get_service_status — the current status of a service and its open incidents.
4. search_service_logs — deterministic sample logs for one service, newest matching lines first.

And two knowledge tools from memory.py:

1. get_runbook — one runbook by its exact slug.
2. search_runbooks — the runbook knowledge base searched by free text.

MCPServer derives each JSON schema from the same type hints and docstrings the ADK function tools already carry; that add_tool loop is quoted in 3.3. MCP. These six are the exact set the gateway allowlist re-lists in 5.2. MCP Gateway.

What is deliberately absent matters as much as what is present. The state-changing actions restart_service and resolve_incident, the long-term memory tools, and the instruction-only skills all stay in the agent process. They depend on ADK confirmation context and audit identity that do not survive translation into stateless protocol messages (4.5. Guardrails, 5.2. MCP Gateway). The MCP surface is therefore idempotent by construction: a compromised or misbehaving client can read incident state but holds no verb that changes it.

How do consumers reach the tools?

Both MCP clients in the cluster dial the governed gateway URL, never the raw service. kagent registers the tools through a RemoteMCPServer in infra/kagent/toolserver.yaml:

apiVersion: kagent.dev/v1alpha2
kind: RemoteMCPServer
metadata:
  name: agentops-tools
  namespace: agentops
spec:
  description: Read-only incident, service, log, and runbook tools through agentgateway.
  url: http://agentgateway.agentops.svc.cluster.local:3000/mcp
  protocol: STREAMABLE_HTTP
  timeout: 30s

The url points at agentgateway…:3000/mcp, not agentops-mcp:8000. That is the deliberate design choice: a kagent consumer discovers the tools through the policy point — the CEL allowlist, rate limit, and fail-closed backend of 5.2. MCP Gateway — not around it.

The BYO agent reaches the same route through one environment variable, AGENT_MCP_URL=http://agentgateway:3000/mcp, which swaps its six local read functions for a single remote McpToolset. Its guarded writes, memory, and skills stay in-process. Two different consumers, one governed path, and no code path where a new client quietly acquires a seventh tool.

Owned by 6.3. Platform Agents for the variable and 3.3. MCP for the toolset swap.

How is the raw MCP service isolated in the cluster?

Steering consumers through the gateway is only half the control; the other half is making the raw service unreachable by anything else. Three transport and network layers stack on the agentops-mcp pod, none of which is the tool allowlist:

1. ClusterIP-only service. Service/agentops-mcp is type: ClusterIP, so port 8000 exists only inside the cluster — there is no external address to dial.
2. NetworkPolicy admitting only the gateway. infra/k8s/base/network-policies.yaml selects the MCP pod and permits ingress on 8000 from agentgateway pods alone:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: mcp-ingress
  namespace: agentops
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: agentops-mcp
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: agentgateway
      ports:
        - { port: 8000, protocol: TCP }

1. No ambient identity. The agentops-mcp ServiceAccount sets automountServiceAccountToken: false, and the pod runs runAsNonRoot (UID/GID 10001), readOnlyRootFilesystem: true, all Linux capabilities dropped, and the RuntimeDefault seccomp profile. That is the same hardening 6.3. Platform Agents applies to every course workload that does not call the Kubernetes API.

The BYO agent's own egress policy has no rule to agentops-mcp:8000; it may reach only the gateway and the collector. So even the agent cannot skip the gateway and hit the raw tools — 6.5. Platform Gateway walks the full egress matrix. The picture is layered enforcement, where the blocked paths matter as much as the allowed one:

flowchart TD
    ADK["ADK agent<br/>AGENT_MCP_URL"] --> GW
    KA["kagent RemoteMCPServer<br/>agentops-tools"] --> GW
    GW{{"agentgateway :3000<br/>CEL allowlist, failClosed (5.2)"}} --> SVC
    SVC["Service agentops-mcp :8000<br/>type ClusterIP"] --> POD
    POD["agentops-mcp pod<br/>read-only rootfs, no SA token"] --> SIX["six read tools"]
    Other["any non-gateway pod"] -.->|NetworkPolicy mcp-ingress denies| SVC
    Write["writes, memory, skills"] -.->|never exposed over MCP| SIX

How does the transport reject spoofed Host headers?

Even a caller that reaches port 8000 must present an expected Host authority, which is the application-layer defense underneath the network policy. It closes DNS rebinding: a hostile page pointing a name it controls at an internal address.

The HTTP apps are built with TransportSecuritySettings(enable_dns_rebinding_protection=True, …), and 3.3. MCP quotes that configuration. The allowlist it enforces is not hard-coded: mcp_server.py reads it from the MCP_ALLOWED_HOSTS environment variable, and the deployment sets exactly the authorities the gateway forwards. That is both the short and the fully-qualified name of agentgateway and of agentops-mcp, each with and without a port.

A request presenting any other Host — a DNS-rebinding attempt, a stray probe — is answered 421 Misdirected Request, the HTTP status for a request sent to the wrong server, before any tool runs. 5.2. MCP Gateway shows the same rejection observed from the gateway side.

def _allowed_hosts() -> list[str]:
    """Return the explicit DNS-rebinding allowlist from CSV or secure defaults."""
    raw = os.environ.get("MCP_ALLOWED_HOSTS")
    if raw is None:
        return list(_DEFAULT_ALLOWED_HOSTS)
    hosts = [host.strip() for host in raw.split(",") if host.strip()]
    if not hosts:
        raise ValueError("MCP_ALLOWED_HOSTS must contain at least one host authority")
    return hosts

- name: MCP_ALLOWED_HOSTS
  value: agentgateway,agentgateway:*,agentgateway.agentops.svc.cluster.local,agentgateway.agentops.svc.cluster.local:*,agentops-mcp,agentops-mcp:*,agentops-mcp.agentops.svc.cluster.local,agentops-mcp.agentops.svc.cluster.local:*

How do reads stay coherent with approved writes?

One disk, two mounts, opposite authority: the agent writes the incident database, and the tool server only reads it.

Both mount the same RWO PVC, agentops-agent-state, at /app/state. fsGroup: 10001 sets the group owning its files to the non-root id both pods run as. The agent owns the writable mount; the MCP container mounts that claim read-only:

volumeMounts:
  - name: state
    mountPath: /app/state
    readOnly: true

A confirmed restart or resolution updates the same SQLite database later MCP calls read, without ever giving the six-tool read service filesystem write authority. The sequence runs in three steps, in the order the diagram below draws them.

1. On a fresh volume the agent initializes /app/state/incidents.db from the immutable image seed at /app/data/incidents.db.
2. The MCP container mounts that published claim read-only, so it can neither change nor create the file.
3. The MCP /healthz probe opens the file read-only and stays 503 until it exists and passes an integrity check.

So the read replica fails closed instead of serving — or worse, creating — state. 6.3. Platform Agents covers the probe contract.

flowchart LR
    A["agentops-agent pod<br/>/app/state read-write"] -->|"UPDATE + audit INSERT<br/>one transaction"| DB[("incidents.db on the<br/>agentops-agent-state PVC")]
    A -.->|"first boot: publish seed atomically"| DB
    DB -->|"read-only mount"| M["agentops-mcp pod<br/>/app/state read-only"]
    M --> P{"/healthz: incidents.db<br/>present and valid?"}
    P -->|no| U["readiness 503<br/>pulled from gateway (fail closed)"]
    P -->|yes| S["six read tools serve"]

This is deliberately a single-node, single-replica SQLite design: an RWO claim binds to one node, and the read/write split is filesystem permissions, not a database protocol. A horizontal deployment needs a network database with a migration and concurrency plan, not a multi-writer filesystem assumption.

How do you verify the in-cluster path?

Forward only the gateway — never the raw MCP service — so the check exercises the governed route:

kubectl -n agentops port-forward svc/agentgateway 3000:3000

Run the MCP list-tools script from 5.2. MCP Gateway against http://127.0.0.1:3000/mcp and confirm exactly the six read tools appear and no write action does. Then inspect both sides of the hop:

kubectl -n agentops logs deploy/agentgateway --tail=50
kubectl -n agentops logs deploy/agentops-mcp --tail=50

The gateway log records the routed tools/call; the MCP log records the read reaching the server. A request that never appears in the MCP log but returns an error at the gateway is a policy decision (allowlist, rate limit, or fail-closed backend), not a server fault.

What proves this page worked?

Confirm six reads through :3000, no writes, and a fail-closed response when the backend is gone. Scale the MCP Deployment to zero, repeat the list-tools request, and confirm the gateway denies rather than returns an empty result:

kubectl -n agentops scale deploy/agentops-mcp --replicas=0
# repeat the list-tools request: the gateway must fail closed (500 / -32603), not return []
kubectl -n agentops scale deploy/agentops-mcp --replicas=1

Restore the single replica before continuing, and wait for readiness — the MCP /healthz stays 503 until it re-opens the agent-published database. This checkpoint tests the governed path and its fail-closed behavior, not zone or PV disaster recovery.

You are done when:

• The list-tools call through http://127.0.0.1:3000/mcp returns exactly the six read tools and no write action.
• The gateway log shows the routed tools/call and the MCP log shows the same read reaching the server.
• With agentops-mcp scaled to zero, the same request fails closed (500 / -32603) instead of returning [].
• agentops-mcp is back at one replica and its /healthz reports ready again.

Continue to 6.5. Platform Gateway when the only way you can reach the six tools is through the gateway.